Assessing Inter-rater Reliability With Heterogeneous Variance Components Models: Flexible Approach Accounting for Contextual Variables

Inter-rater reliability (IRR), which is a prerequisite of high-quality ratings and assessments, may be affected by contextual variables, such as the rater’s or ratee’s gender, major, experience. Identification heterogeneity sources in IRR important for implementation policies with potential to decrease measurement error increase focusing on most relevant subgroups. In this study, we propose flexible approach assessing cases due covariates directly modeling differences variance components. We use Bayes factors (BFs) select best performing model, suggest using Bayesian model averaging an alternative obtaining component estimates, allowing us account uncertainty. inclusion BFs considering whole space provide evidence against components covariates. The proposed method compared other frequentist approaches simulation demonstrate its superiority some situations. Finally, real data examples from grant proposal peer review, demonstrating usefulness flexibility generalization more complex designs.